Papers with Singular Value Decomposition
Adaptive Rank Selections for Low-Rank Approximation of Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Singular Value Decomposition (SVD) or its weighted variants has progressed in compressing language models. |
| Approach: | They propose a binary masking mechanism for optimizing the number of ranks in a differentiable framework. |
| Outcome: | The proposed algorithm achieves much better accuracy than previous SVD and its weighted variants. |
Large Language Model Evaluation via Matrix Nuclear-Norm (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) are computationally intensive due to their O(n3) time complexity with Singular Value Decomposition (SVD). |
| Approach: | They propose a metric to quantify the data compression proficiency of large language models and a convex approximation of matrix rank to capture both predictive discriminability and diversity. |
| Outcome: | The proposed model achieves speeds 8 to 24 times faster than Matrix Entropy for the CEREBRAS-GPT model as models increase from 111M to 6.7B . |
NLoRA: Nyström-Initiated Low-Rank Adaptation for Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Parameter-efficient fine-tuning is essential for adapting large language models (LLMs). However, LoRA suffers from slow convergence and some recent LoRA variants, such as PiSSA, rely on Singular Value Decomposition (SVD) for initialization. |
| Approach: | They propose to introduce a small intermediate matrix between the low-rank matrices (A) and (B) and propose NyströmLoRA (NLoRA) which leverages Nyström-based initialization for SLoRA to improve its effectiveness and efficiency. |
| Outcome: | The proposed approach improves on 5 natural language generation tasks and 8 natural language understanding tasks with minimal parameter overhead. |
Compensate Quantization Errors: Make Weights Hierarchical to Compensate Each Other (2024.findings-naacl)
Copied to clipboard
| Challenge: | Emergent Large Language Models (LLMs) use extraordinary performance and powerful deduction capacity to discern from traditional language models. |
| Approach: | They propose a method that uses weights to compensate quantization error and learnable singular value incremental (LSI) LSI is a technique that helps weights compensate each other conditioned on activation. |
| Outcome: | The proposed method achieves state-of-the-art performance in diverse quantization settings, no matter in weight-only, weight-activation or extremely low bit scenarios. |
SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compression (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for compressing Large Language Models suffer from significant truncation losses. |
| Approach: | They propose a novel method that optimizes singular value truncation in SVD compression . they use dynamic compression ratio allocation to balance the large tuncation loss . |
| Outcome: | The proposed method outperforms current state-of-the-art methods on ten datasets and five models on various scales. |
Dynamic Low-rank Estimation for Transformer-based Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | RankDyna is a matrix decomposition method that can be used to compress Transformer-based language models. |
| Approach: | They propose a matrix decomposition method that enables dynamic rank resource allocation . they say it can outperform current SOTA methods under various parameter budget levels . |
| Outcome: | The proposed method outperforms current SOTA methods under various budget levels . the proposed method is more efficient with higher compression rates . |
Adapting Where It Matters: Depth-Aware Adaptation for Efficient Multilingual Speech Recognition in Low-Resource Languages (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent speech foundation models excel at multilingual automatic speech recognition (ASR) for high-resource languages, but their performance drops substantially on low-resourced languages due to the limited data availability. |
| Approach: | They propose a Depth-Aware Model Adaptation framework that allocates adaptation capacity according to each layer’s role. |
| Outcome: | The proposed framework matches or surpasses state-of-the-art accuracy with 80% fewer trainable parameters and achieves 29% error reduction under extreme data scarcity. |
RanLoRA: Residual-aware Nonlinear Low-Rank Adaptation (2026.findings-acl)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) relying on linear low-rank projections restricts adaptation to linear subspaces, limiting flexibility on complex downstream tasks. |
| Approach: | They propose a nonlinear low-rank Adaptation approach that leverages pretrained weights to decompose them into principal components that are kept frozen and residual components that can be used for task-specific adaptation. |
| Outcome: | The proposed approach outperforms vanilla LoRA and representative variants on commonsense reasoning, image classification, and mathematical reasoning tasks. |
Beyond Uniform SVD: Dual-Level Optimization across Columns and Modules for LLM Compression (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for low-rank decomposition overlook decomposing errors and suboptimal approximation. |
| Approach: | They propose a low-rank decomposition framework that integrates low-level optimization at column and module levels. |
| Outcome: | The proposed framework outperforms state-of-the-art methods and baselines in SVD and pruning. |
HiSVD: Principled Low-Rank Approximation of LLMs via Hierarchical Modeling of Information Capacity and Spectral Structure (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for generating layer importance ignore the fine-grained influence of spectral distribution shape. |
| Approach: | They propose a hierarchical rank allocation framework with two stages to address this gap . they propose SVD-based lowrank approximation that exploits spectral heterogeneity . |
| Outcome: | Experiments show that HiSVD outperforms state-of-the-art methods on LLMs . |
LESA: Learnable LLM Layer Scaling-Up (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for depth scaling-up rely on empirical heuristic rules for layer duplication, resulting in poor initialization and slower convergence during continual pre-training. |
| Approach: | They propose a method for learning latent parameters between layers by concatenating parameters from each layer and applying Singular Value Decomposition. |
| Outcome: | Experiments show that LESA outperforms baseline models with less than half the cost of existing methods. |
GeoRA: Geometry-Aware Low-Rank Adaptation for RLVR (2026.acl-long)
Copied to clipboard
| Challenge: | Existing parameter-efficient methods for RLVR face limitations . low-rank adaptation methods do not account for the distinct optimization dynamics . |
| Approach: | They propose a low-rank adaptation method tailored for RLVR that exploits the anisotropic structure of RL update subspace and extracts its principal directions via Singular Value Decomposition (SVD). |
| Outcome: | Experiments on large reasoning models show that GeoRA outperforms strong low-rank baselines across RLVR settings while showing stronger generalization and less forgetting on out-of-domain tasks. |
Can Spectral-Clipping Enable Better Learning While Forgetting Less for Low-Rank Adaptation? (2026.acl-long)
Copied to clipboard
| Challenge: | In recent years, low-rank adaptation (LoRA) has emerged as a significant paradigm that freezes pre-trained weights and introduces small, learnable adapters instead of fine-tuning the full set of parameters. |
| Approach: | They propose a low-rank adaptation approach that injects parameterized singular components with spectral clipping into the pre-trained model. |
| Outcome: | The proposed method improves performance and retains pre-trained knowledge while preserving the weights of the model. |
Hierarchical-Task-Aware Multi-modal Mixture of Incremental LoRA Experts for Embodied Continual Learning (2025.acl-long)
Copied to clipboard
| Challenge: | Existing continual learning setups for embodied intelligence focus on executing low-level actions, neglecting the ability to learn high-level planning and multi-level knowledge. |
| Approach: | They propose a Hierarchical Embodied Continual Learning Setups (HEC) that divides the agent’s continual learning process into two layers: high-level instructions and low-level actions. |
| Outcome: | The proposed method reduces the forgetting of old tasks compared to other methods, while orthogonally training the remaining parts. |
HyperAdaLoRA: Accelerating LoRA Rank Allocation During Training via Hypernetworks without Sacrificing Performance (2026.findings-acl)
Copied to clipboard
Hao Zhang, Zhenjia Li, Yifan Gao, Xi Xiao, Heng Zhang, Shuyang Zhang, null Xiaoxincc, Bo Huang, Yuhang Wu, Tianyang Wang, Hao Xu
| Challenge: | Low-Rank Adaptation (LoRA) assumes a uniform rank r for each incremental matrix, not accounting for the varying significance of weight matrices across modules and layers. |
| Approach: | They propose a framework that allows for faster convergence of low-rank adaptive models . they use a hypernetwork to prune the outputs of the hypernetworks to generate parameters . |
| Outcome: | The proposed framework accelerates convergence of AdaLoRA by leveraging a hypernetwork. |
Relaxing the Constraints: A Dual-Importance Projection Mechanism for Lifelong Model Editing (2026.findings-acl)
Copied to clipboard
Zhenghai Chen, Senbin Xu, Jiaxi Tan, Xinhua Wu, Yan Zhang, Xiawu Zheng, Shengchuan Zhang, Ke Li, Sicheng Zhao, Liujuan Cao, Rongrong Ji
| Challenge: | Existing knowledge editing methods rely on strict orthogonal projection to preserve previously edited knowledge, but this constraint limits gradient expressiveness, resulting in degradation of model generalization and overall performance as the number of edits increases. |
| Approach: | They propose a method that leverages Singular Value Decomposition to identify critical gradient subspaces and introduces a dual mechanism comprising "accumulated importance" and "projection importance" |
| Outcome: | Extensive experiments on five mainstream LLMs show that the proposed method achieves an average comprehensive performance improvement of 10.36% and effectively maintains the model’s general capabilities on downstream tasks. |
Beyond Variance: Knowledge-Aware LLM Compression via Fisher-Aligned Subspace Diagnostics (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for activation compression are gradient-blind and preserve high-variance dimensions regardless of their impact on factual knowledge preservation. |
| Approach: | They propose a knowledge-aware compression framework that models activation-gradient coupling by directly modeling subspaces. |
| Outcome: | The proposed framework preserves 6–8% more accuracy on knowledge-intensive benchmarks compared to variance-based methods at 50% rank reduction. |